Abstract
Preference-based reinforcement learning (PbRL) has emerged as a promisingapproach for learning behaviors from human feedback without predefined rewardfunctions. However, current PbRL methods face a critical challenge ineffectively exploring the preference space, often converging prematurely tosuboptimal policies that satisfy only a narrow subset of human preferences. Inthis work, we identify and address this preference exploration problem throughpopulation-based methods. We demonstrate that maintaining a diverse populationof agents enables more comprehensive exploration of the preference landscapecompared to single-agent approaches. Crucially, this diversity improves rewardmodel learning by generating preference queries with clearly distinguishablebehaviors, a key factor in real-world scenarios where humans must easilydifferentiate between options to provide meaningful feedback. Our experimentsreveal that current methods may fail by getting stuck in local optima,requiring excessive feedback, or degrading significantly when human evaluatorsmake errors on similar trajectories, a realistic scenario often overlooked bymethods relying on perfect oracle teachers. Our population-based approachdemonstrates robust performance when teachers mislabel similar trajectorysegments and shows significantly enhanced preference explorationcapabilities,particularly in environments with complex reward landscapes.